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PREFACE 



The Research on Evaluation Program is a Northwest Regional 
Educational Laboratory project of research, development, testing, 
and training designed to create new evaluation methodologies for 
use in education. This document 1s one of a series of papers and 
reports produced by program staff, visiting scholars, adjunct 
scholars, and project collaborators— all members of a cooperative 
network of colleagues working on the development of new 
methodologies. 

Evaluation research can be conceived as a question-answering 
process, and Its resulting knowledge as a question-answer 
proposition. The theoretical study of questions suggests four 
practical strategies for undertaking an evaluation study. 
(1) Before identifying the question to investigate, classify the 
questions that can be asked. (2) Before posing the quest 'cn, 
analyze it. (3) Before addressing the question, construct a 
dummy answer. (4) Before stating the answer, state the 
questions. In general, 1t makes pragmatic sense to expend at 
least as much effort on finding the question as on finding the 
answer. 



Nick L. Smith, Editor 
Pape v and Report Series 
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FINDING THE QUESTION FOR EVALUATION RESEARCH 

Evaluation research can be conceived as a question-answering 
process, and the knowledge that results as a question-answer 
proposition. This conception will be left in abrupt summary form in 
order to pass rapidly on to its implications for practice. What can 
be done with it? How might it enhance the practice of evaluation 
research, in respect of asking and answering evaluation questions? 

Four broad strategies can be suggested, emerging from four 
approaches to the study of questions: categorial theory, erotetic 
logic, pragmatics, and epistemol ogy. Phrased as didactic suggestions 
to a docile evaluation researcher--as pieces of advice imploringly 
requested— they run as follows. 

1. Before identifying the question to investigate, classify the 
questions that can be asked. 

2. Before posing the question, analyze it. 

3. Before addressing the question, construct a dummy answer. 

4. Before stating the answer, state the question. 

These will be set forth under headings of identifying the question, 
posing the question, answering the question, and reporting the answer. 

Identifying the Question 

Before identifying the question to investigate, classify the 
questTons that can be askecH ' 

Classification is one useful way to set about finding the 
question for research. There are other ways to find a question — 
curiosity, luck, inspiration, genius--but these are not ways co set 
about finding it. There are "till other ways that questions are 
identified but neither are they ways of identifying one--copying a 
question from the literature, being assigned one, answering someone 
else's question as posed. 

Two relevant concerns that classification satisfies are: What 
might be some novel and original (possibly fruitful and significant) 
questions to investigate? What is the range of questions that ought 
to be considered for study (so that important, obvious, or necessary 
ones will not have been overlooked)? These concerns might be said to 
relate to the field., A third concern relates to the investigator: 
What personal conceptual involvement does the researcher hsve in the 
enterpri se? 

Classification is a conceptual act. It consists in conceiving of 
the kinds of question that can be asked about the domain or phenomenon 
being i nvesti gated (eval uated) . The result is a set of concepts, or 
classes, each constituted by a generic question--of which the 
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uuestion/s formulated for evaluation will represent particular, 
"dividual variants. The generic questions ' -u, J est a ,s h the 
categories of thought about the phenomenon • 1m ; f ' 
rl«55lfv the auestlons 1s to construct a c • • rrameworx tnai 
defines thought and action In that domain the classification of 

feltTons see Dillon, 1984a; on the relation uetween questions and 
categories see Kahn, 1978 and Rescher, 1982.) 

In a categorlal approa:h to classifying questlons-the approach I 
recJtnd (Dillon, 1984a)--al 1 .of the questions w,11 he the ,mp1est 
ones conceivable, each addressing one of the most , undamental aspects 
o the henomenon Imaginable. No more than one que. ticn aspect 
will be Identified, and no aspect left unidentified. The result s a 
set of a 1 ossible (kinds of) questions askable about a 1 possible 
(kinds of) existential aspects of the phenomenon The passes and 
iiiholasses— distinguished tp whatever degree of detail 1s thought 
useful-are arranged to display the relations that are conceived to 
hold among the categories of the phenomenon. 

One such a scheme appears In Table 1 (f rom Dill ». 1984a). He 
should be mindful that it is not an example of what to do. but an 

nstance of the kind of thing that can ha don.. Each class fication 
is different 1n the principle that it uses, in the classes it 

dentifies, in the domain that 1t describes, n d f ^ * nd _ u ^°P e 
nf its description, and in the purposes for which 1t is constr ""^- 
of 1ts o ° es J r ] i P"°"' 1n fact ver y f e W classification schemes available 

for'ou estfont t i f r thes7reasons that Table 1. although a good 
Hassifi cat-ion Is not what is ordinarily thought of as an example 
and Us a so for ?hese reasons that I recommend that each evaluator 
construct a classification for the particular domain and purpose of 
eval uation. 

With such a scheme as in Tablerl it becomes obvious what possible 
m.oetinnc can be asked, about which possible aspects, of the 
. Uenon'under Investigation; and. ^T^lZllTr 
hPina emDhasized and neglected in previous investigat ons le.g., 
rev ew of re earch) or in the complex of current knowledge (e.g., the 
evaluation sponsor's or client's) about the phenomenon. 

Moreover, it becomes easy to locate within the scheme the 
questio/s of interest to investigate, and to determine a Aether 
these questions can be asked, or others must be askea first, and (b) 
in what form the questions are to be asked. 

what is more, the classification scheme reveals in what relation 
any question o? interest stands to.other questi ons-both questions of 
that kind and questions of otherjq nds . 

Finally the scheme defines the set of possible answers to that 
ocH H a the Form of the eventual ans er, and establishes 

q a U 1 *c irTu-^s part of a proposition formec 1 by the 
question-answer pair, and (b) in relation to answers of that kind and 
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other kinds. The scheme thereby provides a way to assess the 
significance of the evaluation undertaking. 

Illustrations 

It would give relief to these points and their readers if some 
illustrations would now be given. Unfortunately, little relief can be 
forthcoming. Remarkably few schemes are available, all of them 
deficient. 

As for categorial schemes, the two best ones known to me are 
those constructed by Rescher (1982) and myself (Dillon, 1984a and 
Table I infra ); the latter has also been applied to analyzing the 
auest ions^posed in the educational research literature (Dillon, 1983 
1984b). These schemes can be used most fruitfully in conjunction with 
a topical scheme for a given domain, such as evaluation, so as to 
identify various kinds of questions that can be asked about various 
kinds of thing constituting the subject under investigation (e.g., the 
program being evaluated). 

To illustrate, suppose the program to be evaluated is the program 
called Program Evaluation in a university. What are the kinds of 
thing that constitute this subject for evaluation? 

An abbreviated classification might al literati vely run: Purposes 
(Why?) e Persons (Who?), Places (Where?), and Processes (What? How?). 
Each of these categories can be elaborated to a degree thought useful. 
First one asks about omitted categories. One does not ask if anything 
h ,, hppn -pft out. but if any kind of thing has been left out. 
fhousanSs of things are left ouTTcan we conceive of things that do 
not fit in?, and, do these things require us to conceive of another 
'category, or of four-five new categories to replace these four so far? 
Ordinarily (for reasons we needn't concern ourselves with here), one 
cannot just add another category to a list once conceived; one 
reconceives and identifies new categories. 

"Cost," for example, is a popular item in evaluation, but need 
not constitute a new category: it can be conceived either as an 
element of all four categories (the cost of attaining this purpose 
- the cost of these faculty, etc.), or as part of the fourth category 
Processes Of course, depending on purpose, an entire classification 
of tMs program Tould'be based upon the characteristic of cost alone. 

Another classification might re-arrange topical categories in 
order to reflect curricular aspects jf the evaluation program: 
Purposes, Teachers (A Teaching), Students (& Learning) Subject-matter 
(ft Materials), Milieux, Implementation, Evaluation, and Proton. 
"Outcomes" is a popular topic in evaluation ^search but need no he 
a category: it can be conceived as an element attaching to all eight 
main categories, or part of the one category, Students (A Learning) 
Th s scheme, here given in outline only, has been used to classify all 
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Table 1 



Category of Question 



A Classification of Research Questions 

Knowledge 1n Question-Answer 



ZERO ORDER 
0. Rhetorical 



NONE 

i 

no knowledge or no answer. 



FIRST ORDER: PROPERTIES 

1 . Exi stence/Af f i rmation-Negation 

2. Instance/Identification 

3. Substance/Definition 

a. Nature 
h. Label 
t. Meaning 

4. Character/Description 

5. f-'unction/Appl icat ion 

a . Modes 

b. Uses 

c. Means 

6. Rationale/Explication 



INDIVIDUAL ATTRIBUTES OF F\ OF £ 

whether _P i s. 

whether this is a/the K 

what £ i s. 

- what makes £ be P. 

- whether names £. 

- what_P or means. 

what £ has. 
what _P does. 

- how F acts. 

- what~£ can do. 

- how _P does it or is done. 

why or how P has a certain attribute. 
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SECOND ORDER: COMPARISONS 

7. Concomitance 

a. Conjunction 
&. Disjunction 



COMPARATIVE ATTRIBUTES OF P-AND-£ 
whether P goes wi th JJ. 

- whether P and 0 are associates. 

- whether V and I' are alternatives. 



8. 


Equivalence 


whether £ is like and wherein. 


9. 


Difference 


wherein £ and £ differ. 


• 


a. Disproportion 
h Subordination 


t 

- whether £ Is more/less than £. 

- whether P is part/whole of ^. 


THIRD 


ORDER: CONTINGENCIES 


CONTINGENT' ATTRIBUTES OF £-AND-£ 


10. 


Relation 


whether £ relates to £. 


n. 


Correlation 


whether £ and £ covary. 


12. 


Condi tional ity 


whether or how if £ then or If £ then £ 




a. Consequence 

b. Antecedence 


- whecher if P then Q, or what X 1f £. 

- whether 1f 2 tnen Z» or wnat X tnen 


13. 


Biconditional ity 


whether or how If P then Q and 1f £ then £. 


EXTRA 


uiujer: utio 


OTHER ATTRIBUTES OR WAYS OF KNOWING P. 


(14.) 


Del iberdt ion 


whether to do and think £. 


(15.) 


Unspec i 1 i ed 


to know £ in other ways. 


(»6.) 


Unclear 


not known. 



the topics of all articles found 1n curriculum journals during one 
year (Dillon, 1984b). 

Now one conjoins the topical scheme with the categorial scheme in 
Table 1, articulating them to classify the kiMs of questions for 
evaluation: Which aspect of which topic to evaluate? 

We can appreciate that any and all of the Table 1 categories can 
be applied to any one or to all of the topical categories, both singly 
and 1n combination. But I do not recommend making a grid and spinning 
out all the hundreds of. possible combinations. Rather, I have 
r~ ommended classifying the kinds of questions that can be asked. 
Then, with this conception of the domain—the department cal led 
Program Evaluation—one can identify the kind of question that is of 
most Interest, and lastly formulate the particular questions of that 
kind. 

As an alternative, where the question 1s already given by a 
client or sponsor— or Faculty Senate or State Legi si ature — I recommend 
construing that question by situating 1t within the appropriate 
categories of topic plus knowledge about that topic. (In the end, the 
department will be cut anyway, but at least one will learn the kind of 
question that had the cutting edge, and the kind of answer that 
enjoyed the decisive edge.) 

Approximations 

The evaluation literature contains some approximate examples jf 
classification. Three are very instructive. 

Making and collating lists of questions— about everything, from 
everybody— seems to be an emergent approach to "generating- 1 questions 
for evaluation. The multiple questions thus generated are then 
"prioritized" for study. One praiseworthy example are the lists 
offered by Smith (1982) as part of his general effort (e.g., Smitt 
1981) to broaden the" range and variety of questions (and methods) used 
in evaluation research, "The heart of an evaluation design is the 
questions asked" (1982, p. 112). 

As ways of "generating" questions about a teacher education 
program, Smith (1982) proposes that the evaluator consider five 
things: the purpose, audience, and focus of the study,, the criteria 
of the program, and general sources of information about the program. 
He lists 10 possible purposes, 23 audiences, 16 foci, 9 criteria, and 
10 sources of information. For each he gives as "examples" some 
particular questions that are "suggested" by the item. The many 
questions thus generated are then prioritized according to pressing 
proqram needs, external mandates for information, and audience 
information needs; final priority is assigned by the purpose and the 
focus of- the evaluation study, 

- A listing-generating approach can be useful hut far less so than 
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classification, which yields greater conceptual range and greater 
practical control of the questions for evaluation. 

First, lists specify multiple individuals (almost certainly of 
one or two kinds), whereas classification identifies multiple ki nds 
for each of which multiple individuals may be formulated. What looks 
like an impressive variety of questions on the nst can actual ly be a 
monotonous multiplicity of individual variants, all of a kind. For 
range and variety, one does better to stop thinking up examples, 
individuals, particulars and start conceiving of kinds, classes, and 
categories. 

Second, items on lists cannot of themselves generate questions. 
Consider "program goals": Wtfat is the question thus generated? 
"Efficiency": What question does that suggest? Any question 
whatsoever.-an infinite number-can be formulated to include the word 
listed; the questions that are listed as examples must derive from 
some other principle or operation. They are appended to the list, not 
generated by it. By contrast, in th&'classif ication approach 
recommended here, each category is itself constituted of a generic 
'question, of which the particular questions are individual and 

interrelated variants. 

or 

If questions must nt -heless be generated by listings, one 
estimates that a compute- would Jo so better than a poor human 
evaluator. And, indeed, computers are being used to do this very 
thing. Morh*ni 7ing Hypothesis Formation (Hajek & Havranek, 1978) 
reports how a program cat led tiUHA (Genera l Unary Hypotheses Automaton) 
automatically lists all observational statements possible from a set 
of da*a. Applied to industry; for instance, data on simultaneous 
overfishing of the generator and motor of di esel -el ectric locomotives 
permitted GUHA to list 197 "important" statements that then "serve as 
a source of hypotheses for further investigations" (p. 380). But what 
does the evaluator do with the 197 questions thus generated, save 
return them to a computer? From start to finish, class fi cation . would 
give greater conceptual range and practical control of the questions. 

A second approximate example also comes from Smith (1980). Here 

he conceives that evaluators involve themselves in four kinds of 

events, each entailing a kind of question which in turn yields a 
corresponding ynd of answer (via a kind of method). r ^ 

1. Policy . What is the desireable state of affairs? The answer 
is a value claim: X's are good. 

2. Research . What is the state of affairs, and why isit that 
way? The answer is a fact claim: X is true; X because Y. 

3. Management . What needs to be done to achieve the desireable 
state? Tfte answer is an action claim: Do X here. 

4. Evaluation . Is this specific event desireable? The answer 
is a valje attribution: This X is good. 
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uhatPver the merits of this scheme as a classification, it 
illusuales two advantages of proceeding by categories rather than by 
individuals. (Interesting to note, this scheme follows upon Smith s 
effort to classify the evaluation literature.) 

First, it makes immediately apparent that there are d | stin ^ 
classes of questions that are di f ferent-i n kind-from most questions 

e 5 f or evaluation. And it gives the generic questions of the e 
other kinds, so as to provoke any number of individual questions or 
?hat kind. II ! conceives of four kinds of question instead of listing 
40U individuals. ~— 

Second, the scheme shows how the answer depends on the question: 
both are of'one and the same kind. Thus, if you formulate even 100 
questions of a kind, you will get answers all of a kind, e.g., X is 
true. If you wish to know, something of another kind, you must 
formulate not just another question, and not just a fluent 
question, but a question of a different kind. That is another way or 
saying that questions and answers proceed within categories of 
thought Only when you first conceive of the categories do you know 
the kinds of knowledge that your questions will yield; and you know 
further ?he other kinds of knowledge that could be yielded by asking 
questions of another kind. 

A third approximate example is David Nevo's (1983) list of ten 
dimensions of the domain of evaluation research, each formulated as a 
question For example, Definition-How is evaluation defined?; 
Func* Sni— What are the functions of evaluation? The scheme could be 
articulated to incorporate missing categories and the questions 
Reformulated to apply to a particular program being evaluated 
I oTq impaired as a classification-some dimensions are topical 
(e a Objects) , some are epi stemol qgical (e.g., Definition), and 
n?h1rs are ohenomenol ogi cal or existential (Functions)-the scheme 

"ove us example of organizing knowledge of a domain in light 
of the kinds of things known via answers to questions of those kinds. 

Nevo presents a conceptualization of evaluation by reviewing the 
litPrature for the answers it gives to the ten questions. Note that 
t he conception had already been constructed by the act of classifying 
'the (Mnds of) questions -/then the individual studies were reviewed by 
the catagoriss thus established. 

So, too', an evaluator can organize his/her understanding of a 
nroaram beinq eval uated-can conceptualize the program-by first 
r^^ifvina the kinds of topical questions that can be asked about it. 
That yields the kinds of topical knowledge that can be had about the 
program: knowledge of topic X, topic Y, topic Z. 

Then a second, categorial classification can he usee J together 
with the topical scheme, specifying thp kinds of q uest ^°" s .^ h ^ " n he 

I a (InA Sh.ic thP rateaories of knowledge that can be had) about 
e c o e Of the s^ sT/int program: Questions A + B + C about aspect 
x! ABC about aspect V, and Z. For example, quest, ons about the 
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nature, and the functions, and the effects of the teaching i,, tie 
program, and of the program's goals, and of the program s staff. That 
yields knowledge of various kinds about things of various kinds that 
constitute the program being evaluated. 

In the end, the merit of a Classification scheme rests on a 
pragmatic cr1 ter1on--br1ef1y, how useful 1t proves to be for the 
purposes Intended. These purposes are defined by the evaluator (and 
sponsor, etc.) 1" the particular evaluation circumstance. Therefore 
the suggestion here 1s not to study classifications or to search out 
some good scheme somewhere. Rather, the didactic suggestion runs: 
Cl assify the questions . The evaluator 1s the one who must do that for 
the domain being evaluated . 

If that 1s done, novel and* original (kinds of) questions are 
likely to appear, as well as the Important obvious ones not to be 
overlooked. Furthermore, the evaluator will be conceptual ly Involved 
in the evaluation task, establishing the framework for thought and 
action, construing the question-answer and determi n1 ng the 
significance of the evaluation study. Classification is a conceptual 
act, and the actor is the evaluator. 

Other approaches 

By contrast to the conceptual approach of classification, what 
miqht be called a "practical and political" way of Identifying 
questions is suggested by Cronbach (1982). In planning the study, the 
evaluator first entertains the widest possible range of questions, and 
then selects among them. Wide range is assured by consulting not only 
the policy-shapers and program sponsors but also social critics and 
scholars, members of minority and elite communities, professionals 
operating the program and citizens served by it. 

It is not enough to derive questions from program goals, since 
these do not provide for unwanted outcomes (p. 221). The evaluator 
anticipates the possible outcomes that "partisans hope for and 
skeptics fear" (p. 210)--as well as the reverse we might add; and 
then anticipates the competing interpretations for each. These rival 
hypotheses suggest research questions" (p. 219). 'he se ectlon among 
questions is then determined by assessing four characteristics, the 
degree of uncertainty about the answer; the promise of reducing that 
uncertainty; the comparative cost of the inquiry; and the leverage 
that the answer would have on policy or operations (pp. ddb-ddb). 

A number of other approaches are suggested for researchers in 
qeneral as they formulate the question for their individual study. 
Campbell, Daft and Hulin (1982) provide, hesitantly, a detailed 11st 
of "proactive behaviors that might enhance problem finding and 
question generation" (p. 116). One strategy that might apply to 
evasion research is to "bre ? k established mindsets" through vanou 
qroup and individual activities such as brainstorming. Another is to 
state the converse of the conventional wisdom on the subject, assume 
the converse to be true, and see what research questions foil ,w. Two 
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ways, examining the pros and cons of each. 

Still other, and slighter, pieces of advice are listed by McGuire 
f 19731 and Webb 1961), and by the authors of any methods text who 
'1st the usual do's-and-don'ts (do ask a significant question; don t 
ask an unclear question...). 

It should be evident that classification serves in all these 
rases as we" as 1t does also in cases where the evaluator is not 
free to ide ti'fy the question but is presented with one to answer or 
i to ne got ateV questions with sponsors. From whatever source, 
the various questions are fitted within and construed by the 
classif cation scheme that the evaluator has taken care to produce 
beforehand--?!! the determined effort to conceive of the enterprise and 
to maintain conceptual control over it. 

r.nnstrulna the oiven question within this scheme permits most of 
the sa^aa^ta es to accrue to this question « . accrue * h. on., 
that the evaluator sole might have identified thereby. And " yieios 
the further advantage of clarifying for all parties the sense of both 
quest n and answer! For Instance, It prevents a ca egor a sw Itch, 
where an answer 1s construed in a category other than that of the 
wnere an answer » rat lon scheme can be used to see to it that 
antwers°of I I • questions of a kind. Then there can 

be no quest on as to the meanlngV the results of the evaluation 
stuSv (The results may nonetheless be rejected, meaningful I and true 
Though they be; that is* a matter of question-answer pragmatics, to be 
addressed by another suggestion.) 

t wis a missx* 



done. 



are being asked. 

Hpnrv Levin observes: "One of the most neglected areas of 
evalu "o'n enTraU? is that of proper identification o f t e proble. 
(1983, p. 34). -Classification is a way to identify the prohiem 
properly. 

louin aoes on- "Before one begins to address the problem, one 
mU st aTte^t 9 To specifywith great clarity and insight the nature of 
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the problem that ought to be addressed" (p. 35). -Classification is 
a way to specify the nature of the problem with clarity and insight. 

Paul Wortman observes: "The first step in the conduct of a cost 
analysis involves the specification of the problem.... Evaluation 
research, however, has been program, rather than problem, focused 
(1983, p. 248). --Classification is a way to give specification to 
the problem and prob.l em- focus to evaluation- 



Samuel Ball observes that Peril No. 6 of program evaluation is 
Having to Answer the Wrong Questions. "Evaluators are expected to 
provide slick, easy answers to slick easy questions (1982, p. 1/1). 
--Class4.£kcati on is a way to pin the question down and make the answer 
stick to. it . 

Classification is a useful way to go about finding and 
formulating the question for evaluation research. Naturally, no one 
need feel called upon to do it that way. 

Posing the Question 
Before posing the question, analyze it . 

Once the question is posed it £akes on a dynamic and force of its 
own in the thought and action of the investigator. Yet the question 
may be impaired or unfit; it must first be examined and vetted for 
inquiry. Then it can be posed in the dress that analysis reveals as 
most properly suiting the investigation. 

The analysis of questions is far easier than their 
classification. Only two things need to be looked at. These may be 
described informally as (sets of) sentences that come before the 
question, and sentences that come after. Those that come after are 
answers, which are the object of the third suggestion discussed in the 
subsequent section. 

As for sentences that come before the question they are of two 
types: the presuppositions of the question, and the presumptions of 
the questioner. 



Presuppositions 

Presuppositions are sentences (expressing propositions that are) 
entailed by the question-sentence. They are sentences that must be 
true for that the question can have a true answer. If these sentences 
are not true, the question is invalid. It cannot be validly posed 
because ic cannot be truly answered. Logicians say in that case that 
the question does not arise . (For precise formulations of theories of 
question-presuppositions, see Belnap, 1969 and Keenan A Hull, 1973.) 

Here are two favorite examples cited by linguists and logicians. 
"Have you stopped beating your wife?" presupposes: 
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a. you have a wife; 

b. you used to beat her; 

c. either you have stopped beating her or you. are still beating 
her. 

"Is the King of France bald?" presupposes: 

a. there is a present King of France; 

b. the King is eitner bald or not-bald. 

Naturally, anyone can go on to ask such a question anyway and then 
go on to get an answer. The answer will not be a true one an no one 
Sill be the wiser, A certain amount of published research reports 
untrue answers to invalid questions, and no one is the wiser--wiser 
neitheV Tor know? ng tte answer nor for not-knowing that the answer is 



untrue . 
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The analytic questions to ask here are simple ones: 

1. What does this question presuppose? 

2. Is that which it presupposes known to be true? 

If the presuppositions are not known to be true- -they are false or 

deterSinate P (not known to be true or false)--then the research 
question cannot be asked. Another question must be chosen for 
eval uation. 

Particularly useful here is a classification scheme that arranges 
its questions in a hierarchy or sequence, for it d spl ays those prior 
nations whose answers serve in turn as presuppositions to the 
'p "esen? "question* of interest. If those prior quest on: . have no been 
answered, then the subsequent q-estions cannot he asked, for, the 
Truth of the prior question-answer proposition provides just the 
condition for asking the next que.tion-to say noth ng of answering 
it . (Th! scheme also exhibits the subsequent questions that may be 
asked in a programmatic evaluation project.) 

Naturally, any researcher can go on to ask the more significant 
and high-level questions without troubling self with the trivial, 
low IPvel ones Indeed, such is the strategy of preference for 
becoming a re earc er of repute. Thus it transpires that attain 
m C o nr 9 of published research addresses the subsequent question to 
which the prior ones remain unanswered and necessarily serve as 
i e erminate and possibly false Presuppositions to an invalid 
question for which an untrue answer has been proposed as the 
conclusion from significant research But that is of no concern to 
anvone because no one knows that. The problem rather is precisely one 
o^not-knowinq that the answer is untrue, howsoever significant. That 
Remains not-known because the very posing of the question had presumed 

-12- 

IS 



the truth of the presuppositions--that is, those sentences whose truth 
is a condition for there to be a true answer to the question. 



The rush towards the frontiers of knowledge is slowed only when 
intelligence begins to filter in from the cognitively depressed 
hinterland, where toil the faceless researchers over their lowly 
questions, left behind by the path-breakers in their ascent to the 
cognitively lofty pursuits. These minor studies from odd, neglected 
lines of research can prove to invalidate the entire series of 
conclusions by force of demonstrating the falsity of the 
presupposition to prior questions in the series. A far more 
sophisticated and easy way is to examine the truth of the 
presuppositions before posing the question. 

Presumpti ons 

Presumptions a^e those sentences (expressing attitudes that are) 
implied by the questioning-act. They are sentences that must be true 
(accurate of the attitudes) for that the question can be genuinely 
posed. If they are not true, th$ question is fraudulent. . It cannot 
be genuinely asked because the questioning-attitude is absent. That 
means at the least that either (a) that question is not being asked, 
but some other; or (b) no question at al 1 is being asked, but some 
other act or performance or exchange is being engaged in. 

Naturally, a good many questions in everyday life and in 
specialized pursuits, not alone in the research enterprise, are 
fraudulent and so are frauriu iently answered. Here too no one is the 
wiser for it. But the evaluator has to know the score, what's going 
to count, what coinage the questioner will accept— counterfeit or not. 

The first (primary) presumption is that the questioner believes 
that the presupposi tion/s to the question 1s true. It may be false, 
but no matter: Asking the question presumes its truth, i.e., comn.its 
the speaker to it as true, or expresses the speaker's belief that it 
is true. The analytic question here is, Does the questioner (0) 
believe that the presuppositions to the question are true? 

Other presumptions follow and may be variously enumerated and 
formulated. They include the following. 

1. 0 believes that there is a (true) answer. 

2. Q does not know the answer. 

3. 0 needs/wants/desires to know the answer. 

4. 0 believes that the respondent can supply the answer. 

5. o is willing to do what it takes to find the answer. 

(Note that in answering the question, the respondent also shares in 
the presumptions: Respondent believes that 0 believes..., etc.) 
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Another formulation, together with examples of questions that 
violate the conditions, is provided by Knight (1967, p. 

J ,n ..nknnwn based on fact . "When you did stop beati ng your 
wife?"; "Wfio is the Isar of Russia?" 

2. a desire to know the unknown . Questions posed merely to 
disturb violate tFfi condition. 

3. faith that such knowledge exist s and can he acquired. "Where 
are the snows of yesteryear r" 

4 courage toacceptthe r.onseouences of attempting to know. 
"What is the purpose Vfe' or otfler questions wnere certaTTi 
answers would be rejected a prion. 

?lSS5l KlnSp Mil.* ?976° Identifies these types of questions: 

1 irrelevant - no answer will have any logical r ,?,i a V™c° r 
be nef; oft^eTtior.r/audlence. As 1n the adage, "What does that 
have to do with the price of tea In China? 

2 trivial - the answer 1s already known (to the questioner). 
"Does 2+2 equal ' 4 ?"; "Are these words written in English? 

3. foolish - the question is known to Javeno true answer. 
"Which of-thTfol lowing exist: unicorns or chimeras? , Are these 
words printed in red or 1n green ink? 

4. dumb - the question has no direct answers v*atsoever ;i "what 
are at leiiTthree truths from among the following: A and R? 

From Keenan and Hull (1973): 

1 vacuous - of zero truth-value; the presupposition is neither 
true nor TaTiiT- "D?d the students who failed get drunk?" (when all 
students passed). 

? oatholoqical - no answer is false. "Which man likes the girl 
he likes? ^ ''Whi^n stu dents came early and didn't erne early? 
From Rescher (1982): 

1 trivial - the presuppositions of the question afford an 
answer'to U. " Does this question have an answer? 

2. premature - the truth-status of th %P r f p u PP° S s ^ i0n «'Wnat is 
~ o^ -t « 
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the learning processes of extra-terrestrial inhabitants of our 
galaxy?" 

3 inappr opriate - every answer is false, since some 
presupposition is false. "How long is a novel?"; "Why is the moon 
made of green cheese?" 

4. absurd - there is no answer at all, every answer is 
self -inconsistent. "Why is that tree inorganic?"; "Is 'No' the 
correct answer to this question?" 

By contrast, a legitimate or proper question is "one whose 
presuppositions are all (known to be) true" (pp. 136-137). 

Approaches to analysis 

As simply though this narrative has rendered it, the task of 
analyzing the question may seem all too fancy and difficult. Indeed, 
theorists of questions do couch their treatments in occasionally 
remote and forbidding language. No/ietneless , these are the people who 
know most and best about questions. Most of the "practical advice 
available comes from people who have not thought much at all about 
questions, and their advice is practically worthless. 

There are, however, a few sources that make some of the same 
points, but in more familiar language, as do the erotetic logicians 
and linguists. For tnose who would seek help in analyzing their 
question for research, here are some readable yet reliable sources 
(still not as informed or sophisticated as the theorists cited). 

A non-technical, old-fashioned kind of logical analysis is 
suggested by the historiographer David Fischer (1970) in an exce lent 
chapter, "fallacies of question-framing." He reviews ten logical 
fallacies that a research question might exhibit, and he richly and 
humorously illustrates each by analyzing the questions in published 
historical studies. For example, to illustrate false dichotomous 
questons, Fischer cites a dozen published titles including: Martin 
Luther— Reformer or Revolutionary?": "The Robber Barons-Pirates or 
Pior.eefbf"; "The Removal of the Cherokre Nation— Manifest Destiny or 
National Dishonor?" 

Other types include the fallacy of many questions ("When did 
racial segregation harden into its el abcrate mol d?" ) ; metaphysical 
questions ("Was the Civil War inevitable?"); and semantical questions 
("Was the political structure of 17th-century America democratic or 
aristrocratic?") . 

"Without questions of the right sort, the historian's empirical 
projects are consigned to failure befc -e they are fairly begun (pp. 
3-4) With Fischer's list in hand, the evaluator can quickly spot the 
fallacies in the proposed question before he/she goes on laboriously 
and unwittingly to produce, as Fischer's hapless historians did, a 
fallacious answer. It is quite common to pose a fallacious question, 
e.g., a falsely dichotomous question. 
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Another old-fashioned kind of analysis 11 P;? asa " t h ^ r ^ t forth by 
th. nhilosoDher Emmet (1968) In a chapter on "asking the right 
question^ He reviews various ways in which questions can he wrong. 

1 merely ver bal issues . "Can a computer think?"; "Are kind 
people'ge nerous?"; "Is Lconom ics a science?"; "Are we ever really 
free?" 

2 suqqesting the answer . "Do you approve of the proposed 
betrayal cTour country t"; " Are you in favor of this doctrinaire 
egali tari anism?" 

3. built-in wrong assumpti ons. "What is it that determines the 
real wort h or value ot an object ?^ "How do mind and body int jl* 

4. h„ m-1n unans-verabilUy . "Is ttn statement J»Thi s statement 
1s faHe'True or false?"; "What is Reality really like? 

"What we must try to do is tq think what it is that the 
questioner want to kno» if we can discover this, even though 
oerhaos only vaguely, it may be possible by a rephrasing of the 
Question So lay bare or to remove the implicit assumptions and make it 
InElllglbla and answerable" (p. 104). At chapter's end Emmet 
provides exercises for analyzing illustrative questions, and at book s 
end he obligingly supplies the correct answers. 

Lastlv there are fields of practical endeavor, such as opinion 

a clear and simple analysis of questions, e<>di ng wi th * c ™£ n " ° T nt 
inn considerations for wording a question properly. The best current 
1 u T- h oZ hly readable, 'richly Illustrated and groun ed in 

-L..„h < : icn,n riuestions. by Sudman and Bradburn (i98<!). meir 
h^nful Im\ M on? for begin ers add up to the unexpected lesson that 

Questions s a complex skill-is drawn by Hargie, Saunders, and 
Dickson (1981) In a nice review on questioni ng. as a social skill in 
numerous practical contexts. 

The docile evaluator can prof U from any of these sources 
anolvinq tbeir ha-d-won lessons, their experience, ski 1 1 , and researcn 
knowledge, to the task of analyzing the questions proposed for 
evaluation research. 

T , an , 1utif . task is not as difficult as the language of analysis 

The i" 111 it It reaui res asking only a couple of ve-y smart 
may seem to ma<e it. it requi res ny ^ Kc4 . flrl !; hops not lie in 
questions about the proposed question. The obstacle, does not ne in 
he task but in the attitude behind it. Even o undertake the tas<, 
the evaluator must be convinced of two odd beliefs. 



-16- 

22 



I 



1. It 1s the easiest thing 1n the world to ask a wrong research 
question, and most difficult to ask a right one— not to ask a 
world-shaking significant question, just to ask an ordinary 
one in the right way. 

2. By far the major effort of research has to go into getting 
the question ri ght--rather than designing the right method 
and getting the right answer. 

The more common beliefs seem to hold that posing the question is of 
course important but nothing much need be done about 1t; what needs to 
be done is to answer the question— especial ly by performing approbated 
methodological and statistical machinations. The suggestion here is 
rather: Before answering the question, and even before posing it (for 
research), analyze 1t. 

It seems a matter of mere common seise to undertake a leisurely 
reconnaissance of the (presuppositional & presumptive) grounds of the 
question before venturing forth upon them with the e> pensive and 
cumbersome train of research. Even Napoleon, of cor.rse, once failed 
to reconnoitre sufficiently and pressed the attack with Milhaud s 
heavy cuirassiers, all of whom- it will be recalled, arrived just at 
the peak of their plan to tumble one after another* Into the sunken 
road that cut unseen across the grounds clearly in face. 

The faithful evaluator 1s not responsible for the grounds but 
only for knowing what grounds he/she must cautiously proceed upon. 
Before posing the question it is useful to scrutinize its grounds. 

Answering, the Question 

Before addressing the question, construct a dummy answer . 

Long before taking that first step towards finding the answer, 
the evaluator constructs a model of the answer. It is not the 
eventual answer that will emerge but a dummy of it. Makirg the dummy 
answer is harder than analyzing the question but easier than 
classifying the questions to ask. The dummy also takes the longest 
tiroe-^and is the most vexing part of the question-answering process. 
For, it involves other people and their various opinions and 
Intentions regarding the question. 

Naturally, here as elsewhere anyone can go right ahead and get an 
answer without troubling to construct a dummy of it. The answer can 
be a true one, too, and also a genuine one. The problem 1s that the 
answer still won't be any good. The evaluator brings the answer in 
*.nd people either ignore or reject 1t. 3o, first make a-dummy answer 
and bring that in: Is this what you're looking for? Is this what you 
want to know? Can you use something like this? What would you do 
with this one? and how about that model over there? 

Here as elsewhere, it is a matter of pragmatics— pragmati cs as a 
the)retical aspect of the study of questioning, and pragmatics as a 
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practical aspect of question-answering. Scholars have "°t completely 
worked out the theoretical aspects of these pragmatics. But the 
practical aspects can be worked out by the pragmatic evaluator. Some 
helpful formulations will be reviewed for various pragmatic : aspects of 
aivinq answers. Each formulation seems to point to the wisdom of 
making a dummy of the answer before the answer makes a dummy out of 
you. J 

Multiple 'answers 

Only in an ideal world is the evaluator likely to be called upon 
to answer a single question, not to say ask a single question by 
his/her lone self. Even were there to be a single question it would 
likely change from start to finish and in between to boot, becoming 
different questions at various stages of the evaluation. And, even 
were there to be a single question remaining one and the same 
throughout, still there are various people who are posing it-probably 
different questions-and still other various people who are awaiting 
the proposed results— probably different answers. 

The evaluator may be faced with giving multiple answers to 
various people asking different questions. Regrettably, the pragmatic 
evalSator is burdened from the outset with finding out from all sorts 
of people such things as: 

1. who is asking which question? 

2. who is going to get the answer? 

3. who wants the answer? 

.-■> 4. what do they already know? 

5. what do they propose to do with the answer? 

6. what answer do they anticipate, and which one will they 
(a) accept and (b) reject? 

It is useful to find these out before setting out to answer the 
Question. The upshot will undoubtedly be to change the question 
identified for research-to abandon it as posed, to modify and 
reformulate it, to multiply its variants, to add and delete some 
others . 

The evaluator accordingly has to fiddle with the research design; 
and if free advice has been followed, there will not have been much of 
a dean already prepared at this stage. Methods will have to be 
refined, resources reallocated, emphases shifted or reversed, plans 
subtly laid to give this and that answer in this and that way to this 
and that person? Make a dummy of the answer before making a mock-up 
of the design. 

Unwelcome answers 

Without constructing a dummy answer, the evaluator will bring the 



-18- 24 



arduously elicited results Into the expectant arena of events 
receive such welcomes as these. 



1. That's obvious! 



2. That's absurd! 

3. That's irrelevant! 

These niceties of reception reveal that the answer has surprisingly 
proven to relate to the audience's assumption-grounds in particularly 
unwelcome ways. Murray Davis (1971, p. 327) specifies these as 
follows. 

1. obvious. Instead of denying, the answer affirms some aspect 
of the audience's assumption-ground. "Husbands often Influence their 
wives' political behavior." It says: "What seems to be the case Is 
1n fact the case; what you always thought was true 1s- really true. 
The audience's response is: "That's obvious!' 



2. absurd. 



Instead of denying some aspect, the answer denies 
the whole^wsimptlon-ground. "Social factors have no feet on a 
person's behavior." It says: "Everything that seems to be the case 
is not the case at all; everything you always thought was true is 
really false." The audience responds: "That s absurd! 

3. irrel evant . Instead of denying or affirming, the answer does 
not speak to any aspect of this assumption-ground at all. fskimos 
are mSre likely than Jews to..." It says: "What ".really rue has 
no connection with what you always thought was true. The audience 
responds: "That's irrelevant!" 

We may guess that these welcomes reveal, not that che answer is 
wrong, but that the question was the wrong one to answer. It may have 
been, logically speaking, a dumb question (Belnap, 1969); ap< 
probably was, pragmatically speaking, a dumb one to answer^/ 
smarter to make first a dummy answer than to answer a dOffST question. 
Then the dummy can proceed to tell everybody everyth^g that^tney 
always wanted to know but were too dumb to ask. 



Far, 



Interesting answers 

As a result, the audience will welcome the 



j 



nswer wTfclf^ "That's 
interesting!" As Davis (1971) defines It, an Mere^ng Proposition 
denies the truth- of some part of the routinely (tJut w^aBVf held 
assumption-ground of the audience. It 1s interesting^ because it 
Jells them some truth that they did not already know, but that some 
truth that they already know 1s wrong (p. 327). 

As an "Index of the Interesting," Davis provides ^classification 
of twelve kinds V propositions about a phenomenon, according to 
logical categories attaching to it or its relation with other 
phenomena. Each proposition is of the form: What seems to be^XJV" 
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realltv non-X (p. 313), where X represents *he characteristic in the 
iTg cal cate or?. For example, here are three of the twa ve 
categories, each illustrated by a proposition from Sigmund Freud. 

m Composition. What seem to be assorted , -heterogenous 
phenomena are in reali ty composed of a single element. Freud proposed 
that the behaviors of children, primitives, neurotics, and adults in 
crowds, as well as dreams, jokes, and slips of the tongue and pen, are 
various manifestations of the same instinctual drives. . 

(?) Gene ralization . What seems to be a local phenomenon is in 
reality a' general phenom enon. Freud proposed that sexual Impulses are 
a ma or influence on the. behavior not only of adults (which was fairly 
obvious) but also of children (which was not so obvious). / 

(3) Co-existence. What seem to be phenomena which cannot efcist 
together are In reality phenomena which can exist together. Freud 
proposed that love and hate are compatible. 

Davis proposes that this Index can be used as a criterion to 
determine whether or not a particular proposition is interesting (p. 
327) Construing propositions as formed of a question-answer pair, we 
can appreciate that Davis has provided a useful classification of 
interesting answers to questions. 

* We mav also appreciate that to give an interesting answer is a 
matter apart, from giving a correct ..answer. The evaluator's audience 
ran reiect the answer's value while affirming its truth. n the . 
audiencl are pro ram advocates and opponents, experts and lay persons, 
anH sn forth whose relevant beliefs and hopes are mutually , 
contradlctori. The assumptions of the one already constitute a. denial 
of the assumptions of theother. Which answer is interesting to. whom? 

Experts for example, will find interesting an answer that 
strike laypersons as obvious. It denies expert assumptions while 

rm?n lay ones. It says: "What everybody, except experts on the 
subiect think is true is in fact true" (Davis, 1971, p. 331). mat s 
interesting! A boring answer tells experts that what they think is 
true is true. To them, That's obvious! 

To give an interesting answer, then, the evaluator must first 
have leaded what each of the various parts of the audience already 
' lid assume with respect to that particular aspect of the 
^non 5 ^ ^'question (e g., the aspects distingu she in 
Davis's classification). Let the dummy go around and find these 
things out beforehand. 



Influential answers 



A related practical formulation has been worked out for 
evaluators by Cronbach (1982), in two chapters on choosing the 
q e tions and providing the answers. In planning to provide the 
answers, the evaluator is urged to face these questions. 
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1. Will each fraction of the audience (a) attend to the answer, 
<<b) understand it, and (c) find it credible? 

2. Will the answers alter the preconceptions of the audience? 

3. Will the answers enrich and elevate the dialogue leading to 
decisions? (p. 11) 

The valence of the answer must be considered in relation to the 
values of the audience. Cronbach points out that in some cases (e.g., 
Laetrile for cancer) a positive answer will have great effect on 
unbelievers but a negative one will leave enthusiasts unmoved. As a 
general case, Cronbach suggests assessing the possible questions by 
the degree to which the anticipated answers promise to reduce 
uncertainty and to exert leverage. "The planner is to ask: How much 
influence is each of the conceivable answers to this research question 
expected to have? 1 " (p. 226). 

We can appreciate that, to answer Cronbach's questions, the 
pragmatic evaluator must discover and weigh the socio-political as 
well as cognitive positions and purposes of the various recipients of 
the answer. Otherwise, the great effort to obtain reliable results, 
even meaningful and true and good ones, will eventuate 1 n the 
frustrated offer of a useless answer. Let the dummy do it nrst. 

Cooperative answers 

At what might be called a technological level, the pragmatics of 
man-machine question-answering have been worked out by Aravind Joshi 
and associates. They point out that giving a direct, logically 
correct answer is sometimes not helpful; cooperative answers are 
called for. And they have produced a computerized query system, 
QO-OP, that gives cooperative answers. 

Cooperative answers are called for in the frequent case of 
discrepancy in the "mutual beliefs" of the user and the system 
(questioner and answerer), regarding the structure and content of the 
data-base. Joshi defines a cooperative answer as both giving a 
truthful and informative response and squaring away the 
discrepancies in mutual beliefs discerned curing the interaction 
(1983 p 237). Varieties of cooperative responses include correcting 
the questioner's false presumption and providing supportive and/or 
suggestive information in addition to that requested. For example, 
when no linguistics courses had been offered: 

Q. Which students got an F in linguistics in Fall 1980? 

Al. None, {correct, direct, but misleading answer) 

A2. I don't know of any linguistics course in Fall 1980. 

(corrective, Indirect response, cooperative * informative) 
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(Note on terminology: As Joshl s.id Kaplan define it, 
Resumption" incorporates part of what others.-1ncluding this 
pappr-label as "presupposition," along with some of what others also 
label, with Joshi and Kaplan, as "presumption. ) 

CO-OP 1s a natural-language query system (Kaplan, 1981) that does 
lust that— gives cooperative answers. It computes and checks 
presumptions 9 defected in the question's linguistic structure; when it 
detects a false or discrepant presumption, CO-OP generates a 
cooperative response rather than the meaningless correct answer 
Since the presumptions of a question can be partial* ordered by an 
entailment P relationship-the failure of some stalls the failure of 
others— CO-OP responds most appropriately by correcting the 
"least-failing set" of presumptions (p. 137), rather than provoke a 
series 67 little corrective question-answers or, worse, a series or 
little correct, direct answers— "stonewalling (p. 130). To 
illustrate stonewalling: 

01. Which students got an F 1n linguistics in Fall 1980? 
Al. Nil. 

02. Did anyone fail linguistics in Fall 1980? 
A2. No. 

03. How many people passed linguistics in Fall 1980? 
A3. Zero. 

04. Was linguistics given in Fall 1980? 
A4. No. 

In general, then, the principle runs that not only must the 
respondent not give a false answer, but also must not e 
nuestioner to infer from the true answer something that the respondent 
ow t be false (Joshi, 1982, p. 190). Not only must the respondent 
correct the questioner's presumptions but also supply additional 
information ^supportive and suggestive) to make the answer 
helpful --both truthful and informative (Joshi, 19&3, p. t-H ) . 

CO-OP is not the only machine that answers quest ions, nor the 
nniv nnp to do interesting things before answering. PLIDIS is used in 
?he y con?rol of ndustri al 'water pol lution in Germany (Berry-Rogg he. 
5 i lh S i iqro^ It is designed to g ve communicatively 
ItZte answers to busy off 1 s wnoare expert about pollution but 

Q ve bo t question-answer system. PLIDIS checks their questions 
ana corrects erroneous assumptions instead of giving a correct, but 
meaningless answer. 

The answer schema implicit in a question is "a meaning struct ure" 
including presuppositions and presumptions (p. 146). For example, the 
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question, "Did Brecht check the samples from Lauxmann 1n Stuttgart in 
1976?" presumes that Brecht is entitled to collect samples, and 1t 2 
presupposes the existence of: the person Brecht, the firm Lauxmann, 
th» location Stuttgart, the year 1976; samples from 1976, firms 1n 
Stuttgart, a firm Lauxmann In Stuttgart, and samples from the firm 
Lauxmann in Stuttgart 1n 1976 (p. 192). PLIDIS corrects false 
presuppositions as 1n this example: 

0. How many toxic elements were found 1n the November sample at 
VW? 

A. There was no November sample. 

It also corrects false presuppositions, and further it clarifies vague 
questions, as 1n this example: 

01. Was there any cyanide in the Lutz sample? 

Al. Lutz unknown, what syntactic class does the word belonq 
to?--noun, adjective,... 

02 It's a name. 

A2. Name of what kind of thi ng?— company, person, place... 
03. Person. 

A3. Lutz 1s not entitled to collect samples, (correcting a 
false presumption) 

We might say that, before giving an answer, PLIDIS takes care to 
discover what the questioner 1s asking, what he/she understands and 
means by the question, and what sort of answer will satisfy not just 
the question but especially the questioner. Whatever the ques tlon, 
what 1s it that the questioner knows (presumes A presupposes) and what 
is 1t that he/she wishes to know (an Informative answer)? 

We can appreciate that, 1f machines can be made to do as much, 
then the pragmatic evaluator, ever harried and burdened, can probably 
wor', out a way to give cooperative answers to even the stupid 
questions imposed on him for evaluation purposes. One way to do that 
is to make a dummy of the answer and bring 1t 1n for « 'J tt1e e ™™ . 
with the questioner—for a dialogue that will clarify the question for 
the questioner, correct his false assumptions, and help him to put 
into words what he really means to ask. Dummy answers are smart. 

Significant answers 

At a theoretical level, the mosc sophisticated formulation of 
question-answer pragmatics is Grewendorf's (1983) concept of 
"oraqmatlcally significant answers." Briefly, an answer is 
significant when It 1s both Informative and useful to the questioner 
in that situation 1n which the question is asked. 
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To be both informative and useful, the answer ™ t0 
account the knowledge and the purposes of the ques W In the 

q uest1on-s1tuat1on. «^*> he r? n q "where 1 s Lutter S 

different answers acco-d1ng to situation (S). Where is Lutter 

Wegner?" 

si In Munich two friends are conversing about taverns and 
another'frl end joins 1n, reeMzes that L A W 1s a tavern, and asks 0. 
Answer : In Berli n. 

S2. In Berlin someone gets into a taxi and says "Take me to L & 
W." The driver asks Q. Answer : Schlueterstr. 55. 

S3 At the Wittenbergplazt in Berlin a pedestrian asks Q. 
Answer :' Take the Underground to the stop BU and then ask again. 

On the corner of Kant and Leibnitz streets in Berlin a 
pedes Man s Q nswer: Go straight ahead, take the first left as 
far as the second traffic-light and it's right there. 

. S5. From a point where one can see L & W a pedestrian asks Q. 
Answer : Over there. 

xH&i:i;rs/rr,^;ri»£ f . ». 

73) Se can Appreciate that all five of the answers given to this 

riijr feirr* tiiT^s 

a ua™ el-and that Si is a different one in each of the five cases. 
That one 1s the pragmatically significant answer. 

el t f1n y din ^"% e h:^iw r He7sh« ; must .^^n, make assumptions 
and estimates as specif by Grewendorf (1983), about. 

(a) the credulity of the questioner (The questioner believes 
what the answerer says to be true); 

(b) the questioner's state of knowledge, in this situation; 

(c) given (b), the questioner's purpose, in this situation; 

(d) given (abc), the value for the questioner, of a given 
possible answer in this situation. v pp. 7b, /aj 

Then answerer compares an< weighs Pf^ 1 ^^^^^?^ 
gives a pragmatically significant one according to this pragmatic 

postulate: 
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Choose among the answers which you think are true that 
one for which the expected/assumed (by you) pragmatic 
significance is greatest, (pp. 79, 80) 

Our cwn pragmatic postulate would suggest that the answerer do these 
things before going out to get the answe. % and, Indeed, before even 
setting cjt the question (as a question for research). This pragmatic 
postulate runs: ^'*- ? 

Before addressing the question, make a dummy of the 
answer. 

This dummy can save everyone a good deal of money, time, effort, and 
tears. Bringing 1n a dummy might even give the evaluator a reputation 
for being smart. 

4 

Reporting the Answer 
Before stating the answer, state the question . 

If research 1s a question-answering process, then Its result is a 
question-answer proposition. In this proposition is represented the 
knowledge that- researcn yields. The meaning of that knowledge is not 
given by the statement that 1s the answer; meaning is located 1n the 
question-answer pair that constitutes the proposition— the knowledge 
proposed by research. 

Aristotle put this eoi stemol oglcal point 1n, as usual, a simple 
and magisterial statement: The kinds of things that we know are as 
many as the kinds of questions we ask; and it is 1n the answers to 
these questions that our knowledge consists ( Posterior Analytics , 
89b). The Implication too is simple: State the results of the study 
in a question-answer proposition. The Implications of not doing so 
are complex and awful. 

It is a familiar fact of language that a question can have 
several answers. It is familiarly overlooked that an answer in turn 
can have several questions. To which question does this statement 
stand in relation of answer? Without knowing that question (not 
another), that answer cannot be known. What will be known in that 
case 1s the proposition that the recipient construes by force of 
devising a question to which the statement may then be adduced in 
answer. 

Such 1s the activity that a recipient or reader of the evaluation 
report necessarily engages 1n— but no; necessarily w1ttingly--m the 
effort to understand the research. What does this study mean? What 
do the results stowr etc. 

And such 1s, consequently, the provenance of some vast and 
hopeless confusion over the research knowledge 1n any given domain or 
on any given topic: A wonderful diversity of questions 1s attached to 
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one and the same given answer. Reviewers then enter to contribute 
riarlfvlna svntheses by the device of reversing the situation. 
Arching to one art the same given question, of their customarily 
Snexoressed formulation, a wonderful diversity of answers; and then 
declaring the one of their choice that best represents the state of 
Snowleogl on that issue~1.e., reversing things once again so as to 
adduce someone else's answer to their question. 

To anticipate the client's need for meanl ng, and to [^estall 
hf./h.r nprnle«ed reaction (But what does it mean? What is it good 
for?)! the equator can do one s1m p1 e thing: Report the results in a 
question-answer proposition. 

It is a very simple matter. First, state the question: "What 
color is the program?" Then state the answer: "The color of the 
program is red " This elementary task takes extraordinary discipline, 
oTonen observed in research reports; and it yields an oddly 
powerful measure of knowledge in a most economical way, traits again 
not often observed in research reports. 

Esoeciallv where evaluator and client have together explored the 

ve^he advantage of construing the answer, which they read or hear, 
in ceras of the question that precedes and complements it in the 
report of the study. 

At the start of the report, the evaluator may set out all the 
assumptions of the question! At the end he sets out ques ion and 
answer together, in question-answer statements. This catecn ticai 

™1 those others who are implicated in it, presumably to their 
betterment. 

We may note that it is not enough to state the question on the 
first rv.no nf the reDort and then tne answer on the .ast page, mere 
IJv I s?i o 'twixt question and answer. And the reader is not the 
7Ze 3 J,l Vr£ hotd of the connection while threading is way 



1 s 

chtthTst^rAnaly^^ 

T±Zl%ir o y r tLTZZZK ZZ*)l?/l^ 52. i. 

r.ot^lven and no answer to the stated question ,s given. 

The answer alone ("results and conclusions") makes no sense. It 
makes sense to state the question before the answer, in a 
question-answer proposition. 
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Conclusion 

In general, it makes good pragmatic sense to expend at least as 
much effort on finding the right question as on finding the correct 
answer. For, the correct answer to a wrong question will be 
rejected— and rightly so— just as an incorrect answer to the right 
question. 

How do you go about finding the question for evaluation research? 
(—so did I ask myself, specifying, "What do I know about questions 
that might be helpful for the practising evaluator?") Some answers 
are: curiosity, luck, Inspiration, genius. But these are not ways of 
going about doing 1t. Four systematic ways are: 

1. Before identifying the question, classify the questions that 
can be asked. 

2. Before posing the question, analyze it . 

3. Before addressing the question, make a dummy of the answer . 

4. Before stating the answer, state the question . 

These strategies are some of the practical fruits of conceiving 
research as a question-answering process, and its resulting^knowledge 
as a question-answer proposition. 
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